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large-scale  System  Optimization:  A  Review 

by 

George  B.  Dantzig 


Mathematical  programming  is  a  generic  term  for  the  related  fields  of  Linear 
Programming,  Network  Flow  Theory,  Integer  Programing,  Convex  and  Non-Linear  Pro¬ 
gramming,  and  Programming  under  uncertainty.  Its  research  has  problems,  parti¬ 
cularly  those  problems  where  random  events  and  decision  events  occur  alternately 
in  successive  stages.  In  problems  where  such  uncertainty  occurs,  what  is  usually 
done  in  formulating  is  to  replace  the  uncertain  elements  with  their  expected  va¬ 
lues  (with  possible  an  added  safety  factor).  It  is  well  known  that  a  plan  based 
on  expected  values  of  its  coefficients  and  constraints  can  lead  to  answers  that 
sure  not  correct.  Although  the  use  of  expected  value  does  not  lead  to  the  best  an¬ 
swer,  it  is  entirely  possible  that  it  could  lead  to  excellent  plans  indistinguish¬ 
able  from  the  optimum  in  the  run-of-the-mill  application.  When  one  considers  in¬ 
stead,  a  direct  attack  on  uncertainty  via  mathematical  programing,  it  inevitably 
leads  to  the  consideration  of  large-scale  systems.  These,  because  of  their  struc¬ 
ture,  have  proven  difficult  of  solution  so  far,  but,  I  believe,  of  intensive  in¬ 
vestigation  in  the  future. 

Mathematical  programming  is  a  term  invented  by  Robert  Dorfman  of  Harvard  a- 
round  1950.  He  felt  that  at  that,  time,  the  fundamentals  of  linear  programming 
were  well  enough  known  that  the  wave-of- the -future  lay  in  the  extension  of  the 
methods  of  linear  programming  into  the  non-linear  programing  field.  Certainly 
we  today,  15  years  later,  feel  this  is  true.  In  the  Calculus,  the  derivative 
(or  first  order  approximation)  plays  a  key  role.  Applied  to  non-linear  inequal¬ 
ity  systems,  it  leads  to  approximation  by  linear  inequality  systems.  Elis  is  one 
way  which  these  extensions  have  taken  place,  and  Illustrates  why  the  various  fields 
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comprised  under  mathematical  programing  are  related.  Here  are  some  other  vays: 

One  attempts  to  extend  the  concept  of  duality  to  non»llnear  system.  Having 
done  so,  one  tries  to  combine  the  combinatorial  power  of  linear  program  with  the 
classical  steepest  descent  processes  to  solve  non-linear  program. 

One  attempts,  as  ve  have  just  noted,  to  reduce  problem  involving  uncertainty 
to  equivalent  deterministic  system  and  to  large-scale  system  with  special  struc¬ 
ture. 

One  tries  to  solve  an  Integer  program  by  replacing  It  with  an  equivalent  lin¬ 
ear  program;  that  Is  to  say,  by  cleverly  building  up  a  set  of  linear  Inequalities 
that  are  both  necessary  and  sufficient. 

In  all  of  these  developments,  one  characteristic  stands  out;  namely  In  one 
way  or  another,  techniques  for  solving  large-scale  systems  play  a  dominant  role. 

Accordingly,  let  us  look  first  at  direct  methods  for  handling  large  problem. 
Aroun4  195^  or  so,  under  the  auspices  of  The  RAND  Corporation,  William  Orchard- 
Hays  produced  the  first  truly  commercial  linear  programming  code.  It  had  many  fea¬ 
tures  that  helped  amateurs  to  get  their  problem  on  the  machine  with  a  reasonable 
chance  of  getting  an  am  ever.  Today,  the  building  of  a  linear  programing  code 
(complete  with  all  the  special  features)  Is  a  major  undertaking  which  Is  expen¬ 
sive  to  produce  amd  to  maintain. 

As  applications  grow,  there  has  been  an  increasing  demand  to  handle  truly  e- 

normous  systems.  The  Russian,  Kantorovich,  in  his  1939  pamphlet,  envisioned  6uch 

a  possibility.  Already,  linear  programming  models  of  Industrial  systems  have  been 

6  4 

solved  with  more  them  10  variables  amd  10  equations.  These  models,  of  course, 
do  not  have  general  matrix  structure  and  it  is  not  likely  that  any  instamce  of  a 
large  practical  problem  will  ever  have  general  structure.  The  reason  is  obvious. 
JUst  Imagine  the  physical  task  alone  of  finding  all  the  coefifi dents  for  a  thou- 
samd  by  ten  thousamd  general  linear  programs  (there  could  be  a  high  as  10^  non- 
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zero  coefficients). 

Fortunately,  large-scale  practical  models  tend  to  have  a  low  percentage  of 
non-zero  coefficients;  in  fact,  under  sometimes  under  1$.  Orchard-Hays'  first 
code  exploited  this  characteristic  by  making  use  of  a  "pricing  vector".  Ibis  made 
inexpensive  the  selection  of  the  pivot  column  directly  from  the  original  non-dense 
data.  The  pivot  column  here  refers  to  one  of  the  stepa  in  the  simplex  method  for 
solving  linear  programs. 

As  systems  have  grown  in  size,  every  advantage  has  also  been  taken  of  the 
characteristics  of  the  improved  computers.  It  has  been  discovered  recently  that 
the  size  of  the  inverse  representation  of  the  basis  in  the  simplex  method  could 
have  an  important  effect  on  running  time.  Therefore,  compact-inverse  schemes  a- 
long  the  lines  first  proposed  by  Harry  Markowitz  of  RAND  have  become  increasingly 
important.  Recently,  two  groups  working  independently,  developed  this  approach 
with  astounding  results.  For  example,  the  Standard  Oil  Company  of  California 
group  reports  running-time  on  some  of  their  typical  large  problems  cut  to  l/4. 

How  to  find  the  most  compact  inverse  representation  of  a  sparse  matrix  is 
still  an  unsolved  problem: 

Conjecture :  If  a  non-singular  matrix  has  K  non-zero  elements,  it  is 
always  possible  to  represent  them  as  a  product  of  elemen¬ 
tary  matrices  such  that  the  total  number  of  non-zero  en¬ 
tries  (excluding  their  diagonal  unit  elements)  is  at 
most  K  .  [incidentally,  the  impirical  schemes  Just 
mentioned  often  have  no  more  than  K  +  10#C  non-zeros 
in  the  inverse  representation.) 

Dynamic  structures  are  interesting  in  themselves,  and  could  have  important 
applications.  One  such  is  the  linear  control  processes  proposed  by  Pontryagin. 


(I  will  speak  of  his  problem  later  in  connection  with  the  decomposition  prin¬ 
ciple.)  As  early  as  1951*,  I  published  a  paper  on  how  to  compact  the  inverse 
representation  of  the  basis  with  a  staircase  structure  (see  figure).  Again  in 
1962,  I  discussed  smother  method  which  permitted  one  to  find  a  compact  inverse 
and  then  efficiently  maintain  this  compactness  in  moving  from  one  iteration  to 
the  next.  There  have  been  several  other  proposals,  all  excellent,  that  seek 
to  apply  the  simplex  method  to  the  full  system  by  compacting  the  inverse.  As 
far  as  I  know,  none  of  these  direct  proposals  have  been  realized  in  computer 
codes. 


1st  period 
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large-scale  systems  have  been  attacked  indirectly  by  means  of  the  decompo¬ 
sition  principle.  Several  codes  have  been  written,  and  some  of  the  recent  ex¬ 
periences  have  been  encouraging.  J.  F.  Benders  in  his  thesis  "ffertltloning  in 
Mathematical  Programming  i960”,  developed  the  dual  of  the  decomposition  prin¬ 
ciple,  and  shows  how  this  approach  can  be  used  to  deal  with  the  mixed-integer 

programming  problem.  Rosen  and  Beale  have  each  proposed  partitioning  methods 
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for  solving  systems  whose  structures  fit  into  the  framework  of  a  common  hori¬ 
zontal  and/or  vertical  border  while  the  remainder  is  a  diagonal  set  of  indepen¬ 
dent  blocks. 


Purely  combinatorial  problems  form  an  important  division  of  mtheaatical 
programing.  They  fall  briefly  into  two  categories.  Hie  first  are  those  prob¬ 
lems  whose  structures  cure  special  —  like  the  transportation  problem  —  or  the 
minimum  number  of  arcs  which  ’’cover”  nodes  in  a  network  (graph).  For  these, 
special  methods  have  been  sought. 

One  of  the  most  tantalizing  problems  of  this  type  has  been  the  travelling 
salesman  problem.  It  is  so  close  to  a  network-flow  type  problem  that  one  would 
hope  to  find  some  easy  representation  of  the  faces  of  its  polyhedral  solution 

set.  So  far,  none  has  been  discovered.  There  is  also  a  close  relation  between 
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covering  problems  and  the  famous  four- color  problem.  The  other  approach  to  com¬ 
binatorial  problems  is  through  integer  programming.  This  was  first  used  in  195^ 
to  solve  a  particular  large-scale  travelling  salesman  problem  by  Fulkerson, 
Johnson,  and  myself.  In  1958,  Gomory  laid  the  foundations  of  this  field  by 
shoving  how  to  systematically  determine  a  necessary  and  sufficient  system  of 
linear  inequalities. 

The  inter-relation  between  large-scale  system  methods  and  integer  pro¬ 
gramming  was  brought  out  in  a  recent  paper  of  Gomory  entitled  "large  and  Non- 
convex  Problems  in  Linear  Programming".  Here,  Gomory  reviews  in  a  unified  man¬ 
ner,  how  the  ideas  of  integer  programming  and  those  of  the  decomposition  prin¬ 
ciple  can  be  combined  to  solve  many  important  applications  such  as  the  paper- 
trim  problem,  multi -commodity  flows  in  networks,  programming  of  economic  lot 
sizes,  etc. 

Integer  programming  methods  are  being  experimented  with  in  a  number  of 
places.  It  seems  likely  that  we  are  nearing  a  threshold,  and  that  we  will  soon 
see  some  excellent  commercial  codes  produced  and  used  successfully  for  certain 
problems. 


I  will  not,  in  this  presentation,  describe  the  developments  in  non-linear 
programming.  Rather,  I  have  chosen  to  illustrate  the  power  of  certain  non¬ 
linear  programming  ideas,  such  as  the  generalized  linear  program  of  Wolfe  to 
an  interesting  problem  in  linear  control  theory. 

But  first,  I  would  like  to  review  the  concept  of  a  Generalized  Program. 
This  differs  from  an  ordinary  linear  program.  Instead  of  coefficients  in  each 

column  being  known,  the  column  P.  may  be  freely  drawn  from  a  convex  set,  C. 

J  J 
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PROBIEM:  Find  Min  XQ  ,  (Xj  £  0,  P^cCj)  ,  P0€C0QECq  such  that 

Vo  +  "  *  Va  ■  «  • 

As  an  example,  consider  the  CONVEX  PROGRAM:  Find  (x^,  xg,  ...,xq)€  C  com¬ 
pact  and  convex  such  that 

*!<*)  £  0 
♦2(*)  £  0 


♦aW  £  0 

♦0(x)  -  Z(Min) 

where  ^  (x )  are  continuous  convex  functions  of  XcC  .  Let  us  assume  an 

x  ■  x°  is  known  such  that  ^  (x°)  <  0  for  i  ^  0  .  It  can  be  shown  that  the 
generalized  program  on  the  following  page  is  equivalent. 
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Price* 


PROBLEM:  Find  Min  Z  and  £  0  ,  ^  £  0  such  that 


!  i  I 


♦>,  (x) 


<  • 


:Vx) 

♦0(*) 


■  i  i 


\+  ^(x  )  X.q  + 


♦»«*°) 

♦0(*°) 


% 


-  1 
-  0 
-  0 


-  0  x_ 


(-Z)  -  0  1 


t> 


The  position  of  an  initial  basic-set  of  columns  is-  indicated  by  heavy  dots. 
The  associated  set  ’of  simplex  multipliers  are  denoted  by  x^  ;  initially, 

x  ■  it®  where  x^  ■  0  for  1^0  and  Xq  ■  -^(x®)  .  The  next  step  is  to  fonn 


the  Iagrangian 


♦(x)-?0(x)*  £  *1*1(x)+*( 


i-1 


for  x  »  xU  and  to  minijuize  4>(x)  for  xeC  .  Notice  that  the  Iagrangian  is 
precisely  what  we  get  if  we  were  to  "price  out"  the  general  column  of  coeffi¬ 
cients  of  the  variable  X  using  the  price  vector  to  form  the  inner  product. 
Thus,  we  wish  to  choose  x  such  that  this  scalar  product  is  minimum. 

Let  x  •  be  the  value  of  the  minimizing  x  .  If  t(x°)  ■  0  ,  then 

is  an  optimum  eolution.  If  #(x®)  <  0  ,  an  extra  column  is  inserted  in  the 
generalized  program  with  coefficients  [l,#^(x^), . . .,$m(x*),$Q(x^)]  and  vari- 
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« 


able  .  The  problem,  restricted  to  those  variables  vhose  columns  have 

known  coefficients,  is  then  optimized  using  the  simplex  method.  Ibis  gives 

rise  to  a  new  set  of  simplex  multipliers  n  ■  it'  .  Hiis  gives  rise  to  a  new 

2  i 

solution  x  ■  x  ,  etc.  At  any  stage,  the  approximate  solution  is  x  -  Zx^x 

using  for  those  which  solve  the  problem  restricted  to  those  vari- 

bles  with  known  coefficients. 

Let  us  turn  to  a  problem  in  control  theory.  Hie  application  of  mathe¬ 
matical  programing  methods  to  solve  control  problems  has  been  studied  by 
Zadeh  and  Whalen,  by  Ben  rosen,  and  others.  I  would  like  to  confine  myself, 
however,  to  Linear  Control  Theory  as  described  by  Fontryagln,  Botlyanskl,  Gem- 
krelidge,  and  Mischenko  in  Chapter  III  of  their  book  on  thl6  subject. 

We  consider  an  "object"  defined  by  its  n  ♦  1  coordinates  x  * 

(£q  ,  ,  ...,1^)  whose  "motion",  described  as  a  function  of  a  "time"  para¬ 

meter  t  ,  can  be  written  as  a  linear  system  of  differential  equations 


dx 

dt 


Ax  +  Bu 


where  u  ■  (u^  ,  Ug  ,...,ur)  is  a  control  vector  that  must  be  chosen  for  each 
t  from  a  convex  compact  set  U(t)  .  Hie  initial  conditions  at  t  ■  0  are 


x°  ■  (0  »  ^  >•••,$£)  >  (fixed)  . 

Hie  terminal  conditions  at  t  ■  T  is  obtained  by  setting 
(2)  xT  -  X  T  +  ZEq 
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where 

xT-  (0,  ^  .  (Elated) 

Eq  ■  (1,0, 0, . . ,  ,  o) 

and  by  requiring  that  u  ■  u(t)  to  be  chosen  such  that 

(3)  Z  Is  minimum 

As  given  in  Chapter  III  of  the  book  "Mathematical  Theory  of  Optimal  Con¬ 
trol  Processes"  by  Pontryagin,  Boltyashii,  Gemkrelidge,  Mischenko,  the  final 
state  may  be  vrltten  in  the  form 

00  -ZE0  +  /T  PT.tB  u(t)dt  -  b 

0 

T  0  tA 

where  b  ■  3T  -  is  a  known  vector,  and  P^  ■  e  matrix  that  may  be  con 

veniently  computed  as  a  function  of  t  .  For  example ,  for  the  case  of  real  dis 
tinct  characteristic  roots  erf  A  : 

n 

(5)  pt  - etA  I MieMt 

0 

where  are  square  matrices  independent  of  t  .  The  latter  fonmxla  for  the 
is  developed  in  "An  Introduction  to  the  Application  of  Dynamic  Programming 
to  Linear  Control  Systems"  by  F.  T.  Smith  in  RAND  Report  RM- 3 526 -PR,  February 

1963. 
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We  may  formally  write  (4)  as  a  generalized  linear  program 


PROBLEM:  Find  Min  Z,  u  £  0  such  that 


(6)  -ZEq  +  Y  u  -  b 

U  ■  1 

where  Y  nay  be  freely  chosen  from  the  convex  set  defined  by 


(7) 


fT 

I  PT_tBu(t)dt 


0 

for  all  possible  choices  of  u(t)  c  U(t)  . 

The  method  for  solving  the  generalized  linear  program  described  earlier 

for  convex  programming  can  be  applied.  For  brevity,  we  emit  the  question  of 

how  to  obtain  the  initializing  basic  set,  except  to  say  it  is  the  analog  of  the 

phase  I  procedure  of  the  ordinary  simplex  method. 

k  * 

As  soon  as  it  ■  it  is  determined  for  iteration  k  we  seek  a  solution  of 
the  sub -problem  such  that  the  inner  product 


Min  it 


f 

0 


/T 

[Min(n^PT-tBu(t  ))dt 

0 


for'u(t)  €'U(t)  . 


It  is  important  to  note  that  for  each  t  ,  n^PT_^B  is  some  known  vector  c*  . 

Thus,  for  each  t  we  must  solve: 

SUB-TOOBLEM:  Find  Min  c^u  for  u  c  U(t)  . 

If  U(t)  is  a  polyhedral  set,  the  sub-problem  is  simply  a  linear  program.  If 
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U(t)  Is  the  same  for  all  t  ,  then  the  linear  programs  are  the  same  for  all  t 
except  for  the  varying  objective  forms  c^u  .  Interesting  enough,  the  sub-pro¬ 
grams  turn  out  to  be  the  same  as  vhat  Pcntryagin  obtains  using  his  maximal  prin¬ 
ciple  . 

A  control  problem  is  an  example  of  a  dynamic  system  which,  is  t  is  treated 
by  the  straightforward  procedure  of  discretizing  the  time,  would  lead  to  a  large- 
scale  computational  problem.  The  generalized  linear  program  (or  decomposition 

principle  approach),  however,  provides  us  with  a  procedure  which  does  not  re- 

* 

quire  the  discretizing  of  the  time  interval. 

Hie  last  twenty  years  have  been  marked  by  the  accelerated  trend  toward  auto¬ 
mation.  Many  believe  that  not  only  simple  control  processes,  but  soon  the  more 
complex  control  processess  will  be  mechanized.  If  so,  whether  we  like  it  or  not, 
decisions  will  be  made  for  us  by  machines.  Whether  or  not  they  will  be  good  de¬ 
cisions  will  depend  on  how  cleverly  we  have  instructed  the  machines.  This  ia 
turn  will  depend  heavily  on  how  clever  we  have  been  in  developing  solution  tech¬ 
niques  for  solving  large-scale  systems. 

To  this  end,  we  have  sketched  several  ideas:  (l)  taking  advantage  of  the 
low  density  of  the  non-zero  coefficients  in  the  original  matrix,  (2)  finding  a 
compact  inverse  representation  of  the  basis  using  the  simplex  method,  4nd  (3) 
making  use  of  the  generalized  linear  program  or  decomposition  principle  approach. 
We  illustrated  the  latter  on  a  linear-control  problem  and  found  that  it  led  to 
the  maximal  principle  with  the  added  bonus,  however,  that  it  can  be  used  to  con¬ 
structively  converge  to  an  optimal  solution. 
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